Energy Based Video Synthesis

This paper describes an optimal method of inserting new frames or recovering missing frames in a video sequence. The method is based on an optimization scheme using graph-cuts that finds the ‘optimal’ frames to be inserted in between two given frames. The core problem is a typical visual correspondence problem between pixels in two or more frames and having formulated the appropriate energy, graph-cuts can be used for optimization. The two frames are assumed to be ‘close’ and the motion of the objects is small. The motion is seen as a set of two dimensional disparities, and the graph-cuts based optimization is able to find these. Once the disparities are found, an intermediate frame can be trivially placed at an arbitrary position in between the two original frames. The advantage of using graph-cuts instead of the typical techniques used in calculating optical flow lies in the global nature of the graph-cuts optimization. The success of our method is shown with synthetic and real image sequences. We show how the method can be extended to insert multiple frames in between the given two frames. One of the immediate applications is generation of synthetic slow-motion sequences. Key–Words: Video, Frame Synthesis, Energy minimization